Autostereoscopic, high-angular resolution, wide field of view (FOV), multi-view displays provide users with an improved visual experience. A three dimensional display that can pass the 3D Turing Test (described by Banks et al.) will require a light field representation in place of the two dimensional images projected by standard existing displays. A realistic light field representation requires enormous amounts of bandwidth to transmit the display data, which will comprise at least gigapixels of data. These bandwidth requirements currently exceed the bandwidth capabilities provided by technologies previously known in the art; the upcoming consumer video standard is 8K Ultra High-Def (UHD), which provides only 33.1 megapixels of data per display.
Compressing data for transmission is previously known in the art. Data may be compressed for various types of transmission, such as, but not limited to: long-distance transmission of data over internet or ethernet networks; or transmission of a synthetic multiple-view created by a graphical processing unit (GPU) and transferred to a display device. Such data may be used for video streaming, real-time interactive gaming, or any other light-field display.
Several CODECS for compressed light-field transmission are previously known in the art. Olsson et al. teach compression techniques where an entire light-field data set is processed to reduce redundancy and produce a compressed representation. Subcomponents (i.e., elemental images) of the light field are treated as a video sequence to exploit redundancy using standard video coding techniques. Vetro et al. teach multiple-view specializations of compression standards that exploit redundancy between the light field subcomponents to achieve better compression rates, but at the expense of more intensive processing. These techniques may not achieve a sufficient compression ratio, and when a good ratio is achieved the encoding and decoding processes are beyond real-time rates. These approaches assume that the entire light field exists in a storage disk or memory before being encoded. Therefore large light-field displays requiring large numbers of pixels introduce excessive latency when reading from a storage medium.
In an attempt to overcome hardware limitations for the delivery of multi-dimensional content in real-time, various methods and systems are known, however, these methods and systems present their own limitations.
U.S. Pat. No. 9,727,970 discloses a distributed, in parallel (multi-processor) computing method and apparatus for generating a hologram by separating 3D image data into data groups, calculating from the data groups hologram values to be displayed at different positions on the holographic plane and summing the values for each position for generating a holographic display. As a disclosure focused on generating a holographic display, the strategies applied involve manipulating fine at a smaller scale than light field and in this instance is characterized by the sorting and dividing of data according to colour, followed by colour image planes and then further dividing the plane images into sub-images.
US Patent Publication No. 20170142427 describes content adaptive light field compression based on the collapsing of multiple elemental images (hogels) into a single hogel. The disclosure describes achieving a guaranteed compression rate, however, image lossiness varies and in combining hogels as disclosed there is no guarantee of redundancy that can be exploited.
US Patent Publication No. 20160360177 describes methods for full parallax compressed light field synthesis utilizing depth information and relates to the application of view synthesis methods for creating a light field from a set of elemental images that form a subset of a total set of elemental images. The view synthesis techniques described herein do not describe or give methods to handle reconstruction artifacts caused during backwards warping.
US Patent Publication No. 20150201176 describes methods for full parallax compressed light field 3D imaging systems disclosing the subsampling of elemental images in a light field based on the distance of the objects in the scene being captured. Though the methods describe the possibility of downsampling the light field using simple conditions that could enhance the speed of encoding, in the worse case 3D scenes exist where no down-sampling would occur and the encoding would fall back on transform encoding techniques which rely on having the entire light field to exist prior to encoding.
There remains a need for increased data transmission capabilities, improved data encoder-decoders (CODECS), and methods to achieve both improved data transmission and CODEC capabilities for the real-time delivery of multi-dimensional content to a light field display.